Cross-Account AWS S3 Bucket Transfer Guide

Overview

This guide provides instructions for transferring data between S3 buckets that are in different AWS accounts.

Prerequisites

• AWS CLI installed and configured
• Source bucket: open-rvfcast
• Destination bucket: rvfcast
• IAM user in source account: open-rvfcast-data

Setup for Cross-Account Transfer

1. Configure Destination Bucket Policy

Apply this policy to the destination bucket (rvfcast):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::SOURCE_ACCOUNT_ID:user/SOURCE_USER_NAME"
      },
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:ListBucket",
        "s3:GetObjectTagging",
        "s3:PutObjectTagging"
      ],
      "Resource": [
        "arn:aws:s3:::DESTINATION_BUCKET_NAME",
        "arn:aws:s3:::DESTINATION_BUCKET_NAME/*"
      ]
    }
  ]
}

2. Update Source Account IAM User Policy

Apply this policy to the IAM user in the source account:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::SOURCE_BUCKET_NAME",
        "arn:aws:s3:::DESTINATION_BUCKET_NAME"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:GetObjectTagging"
      ],
      "Resource": "arn:aws:s3:::SOURCE_BUCKET_NAME/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObjectTagging",
        "s3:PutObjectTagging"
      ],
      "Resource": "arn:aws:s3:::DESTINATION_BUCKET_NAME/*"
    }
  ]
}

Performing the Sync

Option 1: Sync Using Profile with Source Bucket Access

If you have a profile with access to the source bucket:

1. Configure your AWS CLI profile:

   aws configure --profile source-profile
   

2. Run the sync command:

   aws s3 sync s3://SOURCE_BUCKET_NAME s3://DESTINATION_BUCKET_NAME --size-only --profile source-profile
   

Option 2: Sync Using Profile with Destination Bucket Access

If you have a profile with access to the destination bucket:

1. Configure your AWS CLI profile:

   aws configure --profile destination-profile
   

2. Configure source account credentials:

   aws configure --profile source-profile
   

3. Run the sync command with source and destination profiles:

   # If your AWS CLI version supports it:
   aws s3 sync s3://SOURCE_BUCKET_NAME s3://DESTINATION_BUCKET_NAME --size-only --profile destination-profile --source-profile source-profile
   
   # If not supported, you'll need to use Option 1 instead
   

Additional Flags

• --size-only: Compare file sizes only (faster than comparing checksums)
• --no-progress: Hide progress information
• --dryrun: Show what would be transferred without actually transferring

Troubleshooting

If you encounter "Access Denied" errors:

1. Verify both bucket policy and IAM user policy are correctly configured
2. Check for organization-level policies that might restrict cross-account access
3. Ensure the Block Public Access settings aren't interfering with your policies
4. Verify that Object Ownership settings aren't blocking your access

The most common issue is missing necessary permissions in either the bucket policy or the IAM user policy.